{
"cells": [
{
"cell_type": "markdown",
"id": "117fc616-fd5c-45c3-a910-3003f656bf0b",
"metadata": {},
"source": [
"## 6 - Precision in Predictions\n",
"### Figuring out precision and other types of tests"
]
},
{
"cell_type": "markdown",
"id": "3efc371a-93a3-4a11-a27a-4607cf222138",
"metadata": {},
"source": [
"There are only a few examples this week to drive home the idea of precision in predictions, or equivalently, how *un*certain we are of our predictions.\n",
"\n",
"The main take home message is that our predictions will vary in precision, but that the impact of this precision varies with the magnitude of the effect. A large prediction that is imprecise may be of more use than a tiny, precise prediction - as ever, it all depends on the use of the model. Traditionally, p-values will be significant when precision is high, with comes with larger sample sizes. We must not be misled by significance, and should think carefully about what kind of hypotheses are important about our predictions."
]
},
{
"cell_type": "markdown",
"id": "75cde913-c26c-4959-a6fc-821de92d48c3",
"metadata": {},
"source": [
"### Finding precision\n",
"We will see some examples of precision in prediction. First, lets import all the usual packages we need:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "d62fb6a0-1b28-4f65-8c8e-290096251043",
"metadata": {},
"outputs": [],
"source": [
"# Import what we need\n",
"import pandas as pd # dataframes\n",
"import seaborn as sns # plots\n",
"import statsmodels.formula.api as smf # Models\n",
"import marginaleffects as me # marginal effects\n",
"import numpy as np # numpy for some functions\n",
"\n",
"# Set the style for plots\n",
"sns.set_style('whitegrid')\n",
"sns.set_context('talk')"
]
},
{
"cell_type": "markdown",
"id": "293e0a10-0fdd-4667-9b8b-957791e5161b",
"metadata": {},
"source": [
"## Intervention data\n",
"We will load the data for the childhood intervention study to examine prediction and precision in some detail, and explore how to set specific hypotheses to test.\n",
"\n",
"We download it from here: https://vincentarelbundock.github.io/Rdatasets/csv/mlmRev/Early.csv"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "1737dd45-3ea2-4e03-97de-0d502f88c4c9",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Plot\n",
"ax = sns.pointplot(data=p,\n",
" x='term', y='estimate',\n",
" linestyle='none')\n",
"# Add errors\n",
"ax.vlines(p['term'], p['conf_low'], p['conf_high'])"
]
},
{
"cell_type": "markdown",
"id": "a15aaee3-e020-4cce-957c-fd7d3e9c8616",
"metadata": {},
"source": [
"The *p*-value of this difference is significant, as it shows that we'd not expect to see a difference as large as this if the difference between the interventions was zero.\n",
"\n",
"But that's often a bit of a silly hypothesis to begin with. We'd not run the intervention (or study, or wherever we get our data!) if we thought the difference was really *nothing* to start with. \n",
"\n",
"Instead, imagine the scenario that your new intervention is out to beat an existing one. That one showed a difference of 7 units by age 2. Is this intervention better than that? Testing this is simple:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "9fc0bc6a-da86-40cd-8f58-3d673ff4991e",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"shape: (1, 8)
term
estimate
std_error
statistic
p_value
s_value
conf_low
conf_high
str
f64
f64
f64
f64
f64
f64
f64
"b1-b2=6"
3.490294
1.49698
2.331557
0.019724
5.663905
0.556268
6.42432
"
],
"text/plain": [
"shape: (1, 8)\n",
"┌─────────┬──────────┬───────────┬──────┬─────────┬──────┬───────┬───────┐\n",
"│ Term ┆ Estimate ┆ Std.Error ┆ z ┆ P(>|z|) ┆ S ┆ 2.5% ┆ 97.5% │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str │\n",
"╞═════════╪══════════╪═══════════╪══════╪═════════╪══════╪═══════╪═══════╡\n",
"│ b1-b2=6 ┆ 3.49 ┆ 1.5 ┆ 2.33 ┆ 0.0197 ┆ 5.66 ┆ 0.556 ┆ 6.42 │\n",
"└─────────┴──────────┴───────────┴──────┴─────────┴──────┴───────┴───────┘\n",
"\n",
"Columns: term, estimate, std_error, statistic, p_value, s_value, conf_low, conf_high"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Difference\n",
"p = me.predictions(cog_mod,\n",
" newdata=me.datagrid(cog_mod,\n",
" trt=['Y', 'N'],\n",
" age=2),\n",
" hypothesis='b1-b2 = 7') \n",
"# Subtract one from the other, is it equal to 7?\n",
"p"
]
},
{
"cell_type": "markdown",
"id": "0c29a935-55af-41d0-ae71-5d5b5c409d7f",
"metadata": {},
"source": [
"The predicted difference from seven - *not* zero - is about 2.5, and it is not significant. But note the confidence interval - it might be just less than 7, or quite a bit above it. So while it is not statistically significant, this might well be worth pursuing! This is an example of a large prediction with a wide interval. If the null value is zero this is comfortably significant, but if its 7, it is not - but the interval indicates the effect could be rather a bit larger than 7. Circumstance dictates what the appropriate outcome is.\n",
"\n",
"You can test almost any hypothesis with `marginaleffects` with a careful application, and often, the hypothesis of zero is not that interesting!"
]
},
{
"cell_type": "markdown",
"id": "55783f2d-ebd3-4e3e-a8aa-68977540735d",
"metadata": {},
"source": [
"## Testing an interval hypothesis\n",
"We can also express a hypothesis as an interval. \n",
"\n",
"Imagine this somewhat complex scenario:\n",
"- A previous, more expensive intervention produces a change of 7 points.\n",
"- We are tasked with seeing if our intervention is *at least as good as* - i.e., *is equivalent* to the existing one, for children aged 2.\n",
"- The current intervention will replace the existing one, if it can be shown to be equivalent to give an increase of at least 6.5 - 12.5.\n",
"- The developers would like it to be better than 7, but they'll accept anywhere between 6.5 and 12.5 as equivalent.\n",
"\n",
"Testing this kind of 'interval' hypothesis is easy, but a bit confusing philosophically. A significant result means our estimate is *within* the bounds specified, a non-significant one indicating one or both of the bounds are within the confidence interval and cannot be excluded - i.e., the effect could be lower than 6.5 or bigger than 12.5. We do this using the `equivalence` keyword and a list of the lower and upper bounds, in addition to the standard hypothesis:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "b39181dc-a721-480a-98d5-3831f34b4f16",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Plot\n",
"ax = sns.pointplot(data=eq, x='term', y='estimate')\n",
"ax.vlines(eq['term'], eq['conf_low'], eq['conf_high'])\n",
"ax.axhline(6.5)\n",
"ax.axhline(12.5)"
]
},
{
"cell_type": "markdown",
"id": "9ec3c04c-36d6-4fd3-9f04-4661699161ac",
"metadata": {},
"source": [
"This technique can be very useful. Sometimes you may see a small, precise, and significant result. Rather than taking this as evidence, we may test whether the result is within an interval of effects we could consider either important or unimportant (the definition is up to you!), and test *that*. \n",
"\n",
"The downside is that defining these intervals is difficult, requires justification, and can be frowned on by less statistically literate folk."
]
},
{
"cell_type": "markdown",
"id": "1dce75fe-fd73-4dbf-85a4-79c6ff033b06",
"metadata": {},
"source": [
"## One more example\n",
"Lets consider another example, revisiting the beauty and wages dataset. First, we read it in and fit a simple model where we predict hourly wages from a `looks` variable that goes from 1 to 5, and an `exper` variable that counts the persons years of experience in the workforce. This shows a statistically significant effect of looks and experience:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "9b171329-f0f5-4f0b-a68d-30ef8043df53",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
OLS Regression Results
\n",
"
\n",
"
Dep. Variable:
wage
R-squared:
0.064
\n",
"
\n",
"
\n",
"
Model:
OLS
Adj. R-squared:
0.062
\n",
"
\n",
"
\n",
"
No. Observations:
1260
F-statistic:
42.70
\n",
"
\n",
"
\n",
"
Covariance Type:
nonrobust
Prob (F-statistic):
1.15e-18
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
coef
std err
t
P>|t|
[0.025
0.975]
\n",
"
\n",
"
\n",
"
Intercept
2.5094
0.671
3.742
0.000
1.194
3.825
\n",
"
\n",
"
\n",
"
exper
0.0971
0.011
9.018
0.000
0.076
0.118
\n",
"
\n",
"
\n",
"
looks
0.6373
0.188
3.390
0.001
0.268
1.006
\n",
"
\n",
"
Notes: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
],
"text/latex": [
"\\begin{center}\n",
"\\begin{tabular}{lclc}\n",
"\\toprule\n",
"\\textbf{Dep. Variable:} & wage & \\textbf{ R-squared: } & 0.064 \\\\\n",
"\\textbf{Model:} & OLS & \\textbf{ Adj. R-squared: } & 0.062 \\\\\n",
"\\textbf{No. Observations:} & 1260 & \\textbf{ F-statistic: } & 42.70 \\\\\n",
"\\textbf{Covariance Type:} & nonrobust & \\textbf{ Prob (F-statistic):} & 1.15e-18 \\\\\n",
"\\bottomrule\n",
"\\end{tabular}\n",
"\\begin{tabular}{lcccccc}\n",
" & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$> |$t$|$} & \\textbf{[0.025} & \\textbf{0.975]} \\\\\n",
"\\midrule\n",
"\\textbf{Intercept} & 2.5094 & 0.671 & 3.742 & 0.000 & 1.194 & 3.825 \\\\\n",
"\\textbf{exper} & 0.0971 & 0.011 & 9.018 & 0.000 & 0.076 & 0.118 \\\\\n",
"\\textbf{looks} & 0.6373 & 0.188 & 3.390 & 0.001 & 0.268 & 1.006 \\\\\n",
"\\bottomrule\n",
"\\end{tabular}\n",
"%\\caption{OLS Regression Results}\n",
"\\end{center}\n",
"\n",
"Notes: \\newline\n",
" [1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
],
"text/plain": [
"\n",
"\"\"\"\n",
" OLS Regression Results \n",
"==============================================================================\n",
"Dep. Variable: wage R-squared: 0.064\n",
"Model: OLS Adj. R-squared: 0.062\n",
"No. Observations: 1260 F-statistic: 42.70\n",
"Covariance Type: nonrobust Prob (F-statistic): 1.15e-18\n",
"==============================================================================\n",
" coef std err t P>|t| [0.025 0.975]\n",
"------------------------------------------------------------------------------\n",
"Intercept 2.5094 0.671 3.742 0.000 1.194 3.825\n",
"exper 0.0971 0.011 9.018 0.000 0.076 0.118\n",
"looks 0.6373 0.188 3.390 0.001 0.268 1.006\n",
"==============================================================================\n",
"\n",
"Notes:\n",
"[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
"\"\"\""
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Read in looks\n",
"looks = pd.read_csv('https://vincentarelbundock.github.io/Rdatasets/csv/wooldridge/beauty.csv')\n",
"\n",
"# Fit a simple model and examine output\n",
"looks_mod = smf.ols('wage ~ exper + looks', data=looks).fit()\n",
"looks_mod.summary(slim=True)"
]
},
{
"cell_type": "markdown",
"id": "0fe3b330-9219-4b40-80d8-c6b211dd5d7d",
"metadata": {},
"source": [
"As a silly example, lets assume taking up regular exercise, clean eating, and expensive beauty products will shift your looks from a 3 to a 4. Averaging over your years of experience, how much does going from a 3 to a 4 increase your hourly wage?"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "a3fcb25a-4236-4590-883b-350d3590b360",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"shape: (1, 8)\n",
"┌─────────┬──────────┬───────────┬──────┬─────────┬──────┬───────┬───────┐\n",
"│ Term ┆ Estimate ┆ Std.Error ┆ z ┆ P(>|z|) ┆ S ┆ 2.5% ┆ 97.5% │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str │\n",
"╞═════════╪══════════╪═══════════╪══════╪═════════╪══════╪═══════╪═══════╡\n",
"│ b2-b1=0 ┆ 0.637 ┆ 0.188 ┆ 3.39 ┆ 0.0007 ┆ 10.5 ┆ 0.269 ┆ 1.01 │\n",
"└─────────┴──────────┴───────────┴──────┴─────────┴──────┴───────┴───────┘\n",
"\n",
"Columns: term, estimate, std_error, statistic, p_value, s_value, conf_low, conf_high"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Include hypothesis of zero difference, test to reject\n",
"me.predictions(looks_mod,\n",
" newdata=dg,\n",
" by='looks',\n",
" hypothesis='b2 - b1 = 0') # Same as b2=b1 or 4 - 3"
]
},
{
"cell_type": "markdown",
"id": "6d89caee-106b-4c8a-b9fc-42ea873569d6",
"metadata": {},
"source": [
"The difference of about 0.64 is significant. Hold on though, all that exercise, expensive products and good food is a LOT of effort. Will it earn you at least an extra 0.50 an hour?"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "c3229f4c-4041-4af6-8eaf-8103dec5b53a",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"shape: (1, 8)
term
estimate
std_error
statistic
p_value
s_value
conf_low
conf_high
str
f64
f64
f64
f64
f64
f64
f64
"b2-b1=0.50"
0.137269
0.188008
0.730122
0.465316
1.103719
-0.23122
0.505757
"
],
"text/plain": [
"shape: (1, 8)\n",
"┌────────────┬──────────┬───────────┬──────┬─────────┬─────┬────────┬───────┐\n",
"│ Term ┆ Estimate ┆ Std.Error ┆ z ┆ P(>|z|) ┆ S ┆ 2.5% ┆ 97.5% │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str │\n",
"╞════════════╪══════════╪═══════════╪══════╪═════════╪═════╪════════╪═══════╡\n",
"│ b2-b1=0.50 ┆ 0.137 ┆ 0.188 ┆ 0.73 ┆ 0.465 ┆ 1.1 ┆ -0.231 ┆ 0.506 │\n",
"└────────────┴──────────┴───────────┴──────┴─────────┴─────┴────────┴───────┘\n",
"\n",
"Columns: term, estimate, std_error, statistic, p_value, s_value, conf_low, conf_high"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Include hypothesis of a 50-cent difference, test to reject\n",
"me.predictions(looks_mod,\n",
" newdata=dg,\n",
" by='looks',\n",
" hypothesis='b2 - b1 = 0.50') "
]
},
{
"cell_type": "markdown",
"id": "8f9daf70-378a-44aa-bb94-83eed9ae234b",
"metadata": {},
"source": [
"This difference is not statistically significant, so you conclude the conclude there's no evidence it will improve your wages above 50 cents an hour.\n",
"\n",
"Imagine now you say - this increase of a 3 to a 4 is going to be a lot of work.\n",
"\n",
"Anything from an increase of 0.01 to 1 dollar is absolutely not worth it, and so I'll consider that increase equivalent to what I currently earn - all the gym memberships and beauty products will cost me more than that. An increase of 0.01 to 1 dollar is my interval hypothesis.\n",
"\n",
"You can test that hypothesis like so:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "26127749-a31f-40e6-b9cb-6792ce9fc8cb",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"shape: (1, 13)
term
estimate
std_error
statistic
p_value
s_value
conf_low
conf_high
statistic_noninf
statistic_nonsup
p_value_noninf
p_value_nonsup
p_value_equiv
str
f64
f64
f64
f64
f64
f64
f64
f64
f64
f64
f64
f64
"b2-b1=0"
0.637269
0.188008
3.389585
0.0007
10.480386
0.26878
1.005757
3.336395
-1.929341
0.000424
0.026844
0.026844
"
],
"text/plain": [
"shape: (1, 8)\n",
"┌─────────┬──────────┬───────────┬──────┬─────────┬──────┬───────┬───────┐\n",
"│ Term ┆ Estimate ┆ Std.Error ┆ z ┆ P(>|z|) ┆ S ┆ 2.5% ┆ 97.5% │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str │\n",
"╞═════════╪══════════╪═══════════╪══════╪═════════╪══════╪═══════╪═══════╡\n",
"│ b2-b1=0 ┆ 0.637 ┆ 0.188 ┆ 3.39 ┆ 0.0007 ┆ 10.5 ┆ 0.269 ┆ 1.01 │\n",
"└─────────┴──────────┴───────────┴──────┴─────────┴──────┴───────┴───────┘\n",
"\n",
"Columns: term, estimate, std_error, statistic, p_value, s_value, conf_low, conf_high, statistic_noninf, statistic_nonsup, p_value_noninf, p_value_nonsup, p_value_equiv"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Equivalence hypothesis\n",
"eq = me.predictions(looks_mod,\n",
" newdata=dg,\n",
" by='looks',\n",
" hypothesis='b2 - b1 = 0',\n",
" equivalence=[0.01, 1.])\n",
"eq"
]
},
{
"cell_type": "markdown",
"id": "f494e0f4-7f1b-4e66-a5a5-a9da38cc810b",
"metadata": {},
"source": [
"This *is* significant, which indicates the effect *is within the interval*, and thus you can conclude the end result is the same as what you presently earn:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "52ed376c-cd4d-4776-af15-a7d0a9c6cd59",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAlEAAAHHCAYAAACfqw0dAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA5nElEQVR4nO3de1jUZf7/8dcwjgKelRABD+UBMcnDmmSWeEJ/mZUaua6HNrfSb+rm2rey0vK7pFnpqil52E7mYXN1c2vJqzRRv2m6Jm4eQwg8hCdAARUHdID5/eGXKZLD8JlRBng+rstrdz73Pff9nq4LeXl/7rk/JrvdbhcAAAAqxKuyCwAAAKiKCFEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAgFqVXUB1dujQIdlsNnl5ealOnTqVXQ4AAHDC1atXVVhYKIvForCwsFL7EaJuIpvNJrvdroKCAlmt1souBwAAVIDNZiuznRB1E3l5eamgoEAmk0k+Pj6VXQ4AAHBCbm6u7Ha7vLzK3vVEiLqJ6tSpI6vVKh8fH4WGhlZ2OQAAwAkJCQmyWq3lbsVhYzkAAIABhCgAAAADCFEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAAEIUAACAAVUyRJ04cUJdunRRdHR0hd+blpammTNnKjIyUmFhYerbt69ef/11ZWZm3oRKAQBAdVXlQtT58+c1ceJE5ebmVvi9qampevTRR7V27Vp5e3urb9++MpvNWr16tYYNG6Zz587dhIoBAEB1VKVCVEJCgkaNGqWUlBRD73/ppZeUkZGhSZMmKTY2VosWLdKmTZs0cuRInTt3TjNnznRzxQAAoLqqEiHq4sWLmjt3rkaMGKGTJ08qODi4wmPEx8crPj5erVu31uTJkx3XzWazZsyYocDAQG3fvl3JycnuLB0AAFRTVSJErVy5Uu+//76aNGmipUuXaujQoRUeY+vWrZKk/v37y8ur+Me2WCzq16+fJCkuLs7legEAQPVXq7ILcEZAQICmTZumUaNGydvbW0eOHKnwGElJSZKk9u3bl9jetm1bSdLRo0eNF1qK/IJCZeXkuX1cAJXjqq1AH+064Xg97t7WqmMxV15BANwqv6DQqX5VIkQ99thjLo+Rnp4uSWrWrFmJ7f7+/sX6uVPCuct6aDUrXEB1tXzHicouAYAbvT2gqdo0tpTbr0rcznMHq9UqSfLx8Smx3dvbu1g/AACAstSYEGU2X19qN5lMZfaz2+23ohwAAFDFVYnbee5Qt25dSSr1fKm8vOt7lkpbqXJFx+YNdGDmb9w+LoDKcSnPpvvf2uZ4vWNaXzXwLn/pH0DVkHrsR13NK/88yhoTovz9/XXkyBFlZGSU2F60F6pob5Q7mb1MaujDX7BAddXA28LPOFCNnPEq+65VkRpzOy8kJESSSj0Hquh6UT8AAICy1JgQFRERIUn6+uuvVVhY/KuLNpvNcT5U3759b3ltAACg6ql2IcpmsyklJUUpKSmy2WyO6926dVNYWJhSUlI0f/58xwbygoICzZ49W2fPnlXv3r0VGhpaWaUDAIAqpNrtiUpLS9PgwYMlXT99/JePiJkzZ47GjBmj9957T3FxcWrXrp0SEhL0008/KSgoSLNmzaqssgEAQBVT7VaiytKuXTtt2LBBw4cP1+XLl7Vt2/Vv14wdO1br1q0r9SBOAACAXzPZORjppklISJDVapWvry+3CYFq5GKuTZ3/vNnx+sDMgXw7D6hGnP39XaNWogAAANyFEAUAAGAAIQoAAMAAQhQAAIABhCgAAAADCFEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAAEIUAACAAYQoAAAAAwhRAAAABhCiAAAADCBEAQAAGECIAgAAMIAQBQAAYAAhCgAAwABCFAAAgAGEKAAAAAMIUQAAAAYQogAAAAwgRAEAABhAiAIAADCAEAUAAGAAIQoAAMAAQhQAAIABhCgAAAADCFEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAAEIUAACAAYQoAAAAAwhRAAAABhCiAAAADCBEAQAAGECIAgAAMIAQBQAAYAAhCgAAwABCFAAAgAGEKAAAAAMIUQAAAAYQogAAAAwgRAEAABhAiAIAADCAEAUAAGBArcouwFnHjx/Xu+++q3379unChQsKCAjQAw88oAkTJsjX17dCY+3fv1/Lly/Xf/7zH125ckVNmzbVvffeq2eeeUYtW7a8SZ8AAABUJ1ViJergwYMaPny4YmNj5efnpz59+shqtWrZsmUaOXKkcnJynB7ryy+/1KhRo7R161YFBQWpT58+ql27tjZs2KChQ4fq4MGDN/GTAACA6sLjQ1R+fr6ee+45Wa1WzZ49W+vXr9eiRYu0ZcsW9evXT4mJiZo/f75TY+Xm5uq1115TYWGh/vKXv2jDhg2KiYnRV199pSeeeEJXrlzR9OnTb/InAgAA1YHHh6iNGzcqNTVVPXv2VFRUlOO6t7e33njjDfn6+mrdunW6ePFiuWPt379fly5dUrt27TRkyBDHdbPZrKlTp8psNispKUmZmZk35bMAAIDqw+ND1NatWyVJkZGRN7Q1btxY4eHhstls2rFjR7ljmc1mSVJmZqauXbtWrC0rK0sFBQWyWCyqV6+eGyoHAADVmceHqKSkJElSSEhIie1t27aVJB09erTcscLCwtSkSROdP39e//3f/61jx44pLy9PBw4c0KRJkyRJTzzxhGrXru2m6gEAQHXl8d/OS09PlyQ1a9asxHZ/f/9i/cri4+OjmJgYTZ06VZs3b9bmzZsdbd7e3oqOjtZvf/tbN1RdnN1uV0FBgdvHBVA5Cn/181xYUKCCAo//NykAJ9ntdqf6eXyIslqtkq6HnJIUXS/qV57bb79dDz/8sD788EN16NBBAQEBSkpKUmpqqlasWKGOHTsqLCzMPcX/n9zcXO3fv9+tYwKoPFeuFRZ7fejQIdWtTYgCahqPD1Fms1mFhYUymUxl9nMmNZ49e1ZjxoxRVlaWPvjgA/Xs2dPx3o8//lhz5szRuHHjFBsbq+bNm7ulfgAAUD15fIiqW7eusrOzlZubW2J7Xl6epOu36sqzYMECnTp1Si+//LIjQEmSyWTSE088ocOHDys2NlYrVqzQyy+/7J4P8H+1lbanC0DVcynXJn0e53gdFhamBj6WSqwIgDslJiaWmjt+yeNDlL+/v7Kzs5WRkaEWLVrc0F60F6pob1RZdu3aJUm6//77S2zv06ePYmNjdfjwYRcqvpHJZHJ8MxBA1edlLvzVazM/40A1Ut7dryIefxO/aAUnOTm5xPai686s9BSdJVWrVsnZsegvwfz8/ArXCQAAahaPD1ERERGSpE2bNt3QlpWVpT179shisahXr17ljtWmTRtJP5899Ws7d+6UJIWGhhotFwAA1BAeH6IiIyMVGBionTt3as2aNY7reXl5mj59uqxWq6KiouTn5+dos9lsSklJUUpKimw2m+P66NGjJUkxMTHau3dvsXn+8Y9/6NNPP5XFYtGYMWNu8qcCAABVncfvifL29tabb76p8ePHKzo6Wp9++qmCg4P1/fffKz09XR07dtTzzz9f7D1paWkaPHiwJCkuLk7BwcGSpMcee0yHDh3S3//+d40ZM0ZhYWEKCAhQcnKyjh8/LovFotmzZzsO8AQAACiNx4coSQoPD9f69esVExOj7777TsnJyQoODlZUVJSefPLJCj2mJTo6Wr1799Ynn3yiw4cPKyEhQY0bN9aQIUP01FNPcSsPAAA4pUqEKElq3769Fi1a5FTf4OBgJSYmlto+YMAADRgwwF2lAQCAGsjj90QBAAB4IkIUAACAAYQoAAAAAwhRAAAABhCiAAAADCBEAQAAGECIAgAAMIAQBQAAYAAhCgAAwABCFAAAgAGEKAAAAAMIUQAAAAYQogAAAAwgRAEAABhAiAIAADCAEAUAAGAAIQoAAMAAQhQAAIABhCgAAAADCFEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAAEIUAACAAYQoAAAAAwhRAAAABhCiAAAADCBEAQAAGECIAgAAMIAQBQAAYAAhCgAAwABCFAAAgAGEKAAAAAMIUQAAAAYQogAAAAwgRAFABaRfytOq3SeKXVu1+4TSL+VVTkEAKk2tyi4AAKqCQ6cuatk3Kdp0+JzyC+3F2uZtTtLCLT9qUKcA/VfvNgoLblhJVQK4lQhRAFCO1f8+qdc+P6xfZadi8gvt2njwrL48dFbRj3TSmHta3boCAVQKQhQAlGH1v09qxmeHne5faJdmfHZYJpM0OpwgBVRn7IkCgFIcOnVRr33ufID6pVc/O6xDpy66uSIAnoQQBQClWPZNSpm38MpSaJeWf5Pi3oIAeBRCFACUIP1SnjYdPufSGF8dPse39oBqjBAFACX414EzN3wLr6LyC+3614EzbqoIgKdx28byixcvavfu3Tp27JguX76sadOm6erVqzpw4IB69OjhrmkA4JZIychx0zhX3DIOAM/jcoiy2+1avHixPvroI+Xl/bxsPW3aNJ06dUq///3v1aVLF7377rtq0qSJq9MBwC1x5WqBm8bJd8s4ADyPy7fzXnzxRS1dulS5ublq1KiRfHx8HG3Z2dmy2+3av3+/xo4dq9zcXFenA4Bbom4ds5vG4SQZoLpyKURt3rxZsbGxatKkid577z3t3r1bHTp0cLT/5je/0Zo1a9S0aVMdO3ZMK1eudLlgALgV2txWz03j1HXLOAA8j0sh6u9//7tMJpP+8pe/6P777y+xz29+8xstXLhQdrtdmzZtcmU6ALhlHu4cqFpeJpfGqOVl0sOdA91UEQBP41KIOnz4sJo3b6577rmnzH7du3dXUFCQTpw44cp0AHDL+Dfw1qBOAS6N8f86Bci/gbebKgLgaVwKUVarVY0aNXKqb5MmTZSfb3yD5fHjx/X888+rb9++uuuuuzRw4EAtWLBAVqu1wmNZrVbFxMTooYceUufOndW1a1eNGjVKX331leH6AFQ//9W7jYwuRnmZpAm927i3IAAexaUQ5efnp5MnT8puL/ssFZvNphMnTsjPz8/QPAcPHtTw4cMVGxsrPz8/9enTR1arVcuWLdPIkSOVk+P8V5HPnz+vxx57TIsXL1ZmZqbuu+8+hYSE6D//+Y+mTJmiFStWGKoRQPUTFtxQ0Y90MvTe14d2UlhwQzdXBMCTuBSievToIavVqr/97W9l9vv44491+fJlde/evcJz5Ofn67nnnpPVatXs2bO1fv16LVq0SFu2bFG/fv2UmJio+fPnOz3e9OnTlZycrEGDBmnr1q169913tXbtWn344YeyWCx6++23deYMh+MBuG7MPa00e1gnp1ekvEzS7GGdePgwUAO4FKLGjRsnLy8vvfXWW1q5cqWysrKKtV+4cEELFy7U/Pnz5eXlpTFjxlR4jo0bNyo1NVU9e/ZUVFSU47q3t7feeOMN+fr6at26dbp4sfwHfR48eFDbt29Xy5YtNXfuXNWpU8fRdu+992r48OEKCAjQ4cPGHjgKoHoaHd5Kn0+6T0Pual7qZvNaXiYNuau5Pp90HwEKqCFcOsCkQ4cOeuWVVzRr1izNmTNHc+bMcbT17NlT2dnZkq4fyDllyhTdddddFZ5j69atkqTIyMgb2ho3bqzw8HBt27ZNO3bs0JAhQ8oc68svv5QkjR07tliAKhIdHV3h+gDUDGHBDRUzqpvSL+VpXXyq5m1OcrQ9P7C9RtzdQv712UQO1CQunwI3evRotWjRQvPnz9fRo0cd14tWpVq1aqUpU6Zo8ODBhsZPSrr+F1VISEiJ7W3bttW2bdt09OjRckNU0QpTly5dZLVatWnTJh06dEgFBQUKCwvTkCFD5O3t/r8E7Xa7Cgrcc/oxgMrVtK5Fo3u0KBaiRvdooQY+Fn7OgWqivL3eRdxylG7v3r3Vu3dvnT59Wj/++KMuX74sHx8f3X777WrTxrVvp6Snp0uSmjVrVmK7v79/sX5lKTpiISsrS0OGDNHp06cdbWvXrtXSpUu1bNkytWvXzqWafy03N1f79+9365gAKs+Va4XFXh86dEh1a/M8d6CmcSlE7d27V/Xr13ecUh4UFKSgoKAS++7cuVMnT57U6NGjKzRH0REGpa0QFV135qiDom/xPf/88woMDNTq1asVGhqqU6dOad68edqxY4eefvppffHFF6pXzz2nFQMAgOrJpRA1duxYde/eXatXry6378KFCw2FKLPZrMLCQplMZX81xpmlt6tXr0qSateurZUrV6phw+tfP+7QoYOWLVumYcOGKSkpSWvXrtVTTz1VoTrL4uPjU+rtSABVz6Vcm/R5nON1WFiYGvhYKrEiAO6UmJjo1PN+nQ5ROTk5N3z7TpLy8vKUmppa6vvsdrtOnz6tY8eOOX2P8Zfq1q2r7OzsUj9MXl6eJBV78HFpvL29deXKFQ0dOtQRoIrUqlVLI0eOVHR0tHbv3u3WEGUymWQ2u+dhpgAqn5e58FevzfyMA9VIeQs3RZwOUVeuXNFDDz3kWM0pmuTIkSMaOHCgU2N06dLF2ekc/P39lZ2drYyMDLVo0eKG9qK9UEV7o8ri5+enK1euKDg4uMT2ouslhUUAAIBfcnonZLNmzTRu3DjZ7XbHH0nFXpf1p3nz5poxY0aFCyy6DZacnFxie9F1Z26XFfVJS0srsT0jI0PS9UfUAAAAlKVCe6ImTpzoOPDSbrdrwIABCgsL08KFC0t9j5eXl3x9fW+4feasiIgIxcbGatOmTRoxYkSxtqysLO3Zs0cWi0W9evUqd6w+ffpo8+bN+vLLLzVp0iRZLMX3MHzzzTeSrp/EDgAAUJYKfSfXYrE4voEXHBysu+++W507d3ZcK+lP8+bNDQco6fohm4GBgdq5c6fWrFnjuJ6Xl6fp06fLarUqKiqq2HP5bDabUlJSlJKSIpvN5rg+ePBgBQcH68SJE4qOji7Wtn79em3atEkNGzbUo48+arheAABQM5jsRnZ732J79uzR+PHjlZeXpzvvvFPBwcH6/vvvlZ6ero4dO2rVqlXFjiQ4deqU+vfvL0mKi4srtgfq8OHDeuqpp5SVlSV/f3917txZJ0+eVFJSkurUqaOFCxeqX79+bqk7ISFBVqtVvr6+Cg0NdcuYACrfxVybOv95s+P1gZkD1ZBv5wHVhrO/v91+OlxhYWGxP9euXdOlS5eUlJSk5cuXGxozPDxc69ev16BBg3TmzBlt375d9evX18SJE28IUOXp1KmTYmNjNXbsWNWuXVvbt29XVlaWHnzwQa1bt85tAQoAAFRvLq9EffPNN1q8eLGSkpJ07dq1cvsnJCS4Ml2VwkoUUD2xEgVUb87+/nbpsM3Dhw9r4sSJKigoKPcMqFq1aqlr166uTAcAAOAxXApRK1euVH5+vtq1a6ennnpK3t7emjJligYNGqTf/va3OnfunDZs2KD4+Hh1795dK1ascFPZAAAAlculEBUfHy+z2axFixbp9ttvlyQ1b95cqampuvfeeyVJw4YN07PPPqstW7Zo48aNevDBB12vGgAAoJK5tLH8woULCgwMdAQo6fpz6H65P8pkMunll1+WJG3YsMGV6QAAADyGy9/Oa9SoUbHXrVu3VkFBgY4fP+64FhgYqFatWikxMdHV6QAAADyCSyGqadOmjkelFCk6k+nXj2kpepAwAABAdeBSiOrUqZPS0tK0Y8cOx7U77rhDdrtde/fudVzLy8vTTz/95NLJ5QAAAJ7EpRD1yCOPyG6369lnn9XcuXOVn5+vrl27qmHDhvrHP/6hzz77TElJSZoxY4YuX75cbO8UAABAVeZSiOrfv78efPBB5ebm6uOPP5bZbJa3t7d+97vfKT8/Xy+//LIeeeQRbdy4USaTSY8//ri76gYAAKhULh1xIEl/+ctf1KtXL3377bcymUySpMmTJystLU2fffaZ7Ha7zGazxo0bp4EDB7pcMAAAgCdwOURJ0vDhwzV8+PCfB61VS3PmzNHUqVN15swZtWzZUk2aNHHHVAAAAB7BLSGqNP7+/vL397+ZUwAAAFQKt4Soa9eu6dixY8rJySn3GXp33323O6YEAACoVC6HqAULFmjlypXKy8srt6/JZNIPP/zg6pQAAACVzqUQtWLFCi1fvtzp/uWtUgEAAFQVLoWodevWyWQyafjw4Zo0aZL8/f1Vq9ZN3WYFAADgEVxKPKmpqWratKlef/11eXm5/Bg+AACAKsOl5FOvXj3ddtttBCgAAFDjuJR+7r77bh0/flxXrlxxVz0AAABVgkshauLEiSooKNCf//xnNo0DAIAaxaU9UR06dNA777yjyZMna//+/erVq5eaNGniePxLSSZPnuzKlAAAAB7BpRCVk5Ojv/71ryosLFRqaqrWrl1b7nsIUQAAoDpwKUQtWLBA+/fvlyTddtttCgwMlMVicUddAAAAHs2lEBUXFyeTyaQ33nhDw4YNc1dNAAAAHs+ljeWZmZlq1aoVAQoAANQ4LoUof39/mc1md9UCAABQZbgUogYNGqRjx47pyJEj7qoHAACgSnApRD3zzDNq2bKlnnnmGX311VccugkAAGoMlzaWR0dHq0WLFtq5c6emTp0qk8mk+vXry8fHp8T+JpNJ27Ztc2VKAAAAj+BSiPrXv/7l+P92u112u10XL17UxYsXS+xf1iGcAAAAVYlLIWrOnDnuqgMAAKBKcSlEcbQBAACoqVzaWA4AAFBTOb0StXv3bklSt27dVKdOnWLXKqJnz54Vfg8AAICncTpEjRs3Tl5eXtq4caNuv/12x7WKbBY3mUz64YcfKl4lAACAh6nQnqjCwsIbrtntdqffX5G+AAAAnszpEHX06FGnrgEAANQEt3Rj+YULF27ldAAAADeNSyGqf//+mjp1qlN9R44cyZEIAACg2nApRJ0+fVrp6enl9issLFRGRoaysrJcmQ4AAMBjOL0nKjk5WTNnzrzhelJSkkaPHl3q++x2u9LS0nTmzBkFBgYaqxIAAMDDOB2i2rZtK29vb3377beOayaTSZcvX9a+ffucGmPMmDEVrxAAAMADVeiIg1dffVVffPGF43VMTIwCAwM1fPjwUt9jMplUt25dhYaGKjw83HilAAAAHqRCIap169aaPHmy43VMTIyaN29e7BoAAEBN4NIDiOPi4hyPgAEAAKhJXApRQUFBpbbl5eVp165dKiwsVPfu3dWoUSNXpgIAAPAoLoUoSUpLS9PSpUsVGBio8ePHS5JSUlI0btw4ZWRkSJJ8fHw0a9YsDR482NXpAAAAPIJLISozM1MjRoxQenq6+vTp47j+2muvKT093bGpPCcnRy+++KJCQkLUpk0bV2sGAACodC4dtvnxxx8rLS1NLVu21G9/+1tJ0smTJ7Vv3z6ZzWZ98sknio+P1/jx45Wfn68VK1a4o2YAAIBK51KI+uabb1SrVi198MEHjpWo7du3S5K6deumLl26SJL++Mc/qkGDBvr3v//tynQAAAAew6UQlZqaqtatWys4ONhxbdeuXTKZTLr33nsd1ywWi4KDg516RExpjh8/rueff159+/bVXXfdpYEDB2rBggWyWq2ufARJ0ltvvaWQkBAtXrzY5bEAAEDN4FKIKigoUO3atR2v8/PztXfvXklSjx49ivXNzc2VyWQyNM/Bgwc1fPhwxcbGys/PT3369JHVatWyZcs0cuRI5eTkGP4M3377rT766CPD7wcAADWTSyEqKChIp0+fls1mkyTt3btXVqtVdevWddzKk65/gy81NVXNmzev8Bz5+fl67rnnZLVaNXv2bK1fv16LFi3Sli1b1K9fPyUmJmr+/PmG6s/MzNS0adNkt9sNvR8AANRcLoWosLAwXbp0SfPmzdPRo0e1cOFCmUwmRUREyGw2S5IuXLigF154QQUFBerZs2eF59i4caNSU1PVs2dPRUVFOa57e3vrjTfekK+vr9atW6eLFy9WeOxXXnlFWVlZ6tatW4XfCwAAajaXQtTTTz8tb29vrVy5UsOGDdOBAwdkNpv19NNPS5Li4+MVERGhvXv3qn79+vrDH/5Q4Tm2bt0qSYqMjLyhrXHjxgoPD5fNZtOOHTsqNO6aNWu0bds2TZo0SZ06dapwXQAAoGZzKUTdcccd+vDDDxUWFqbatWurffv2Wrp0qTp06CBJ8vf3V35+vtq1a6dPPvmk2AZ0ZyUlJUmSQkJCSmxv27atJOno0aNOj/njjz/qrbfeUrdu3TRhwoQK1wQAAODyieVdu3bVunXrSmwLDg7WZ5995ghVRhR9o69Zs2Yltvv7+xfrV56rV6/queeek8Vi0dy5cx23HW8mu92ugoKCmz4PgFuj8Fc/z4UFBSoocOnfpAA8iLN7pV0OUUV27Nihbdu26dixY7p8+bI+/fRT5eTkaMuWLWrWrJkaN25saNyiIwy8vb1LbC+67uxRB2+//baSkpL01ltvGVoZMyI3N1f79++/JXMBuPmuXCss9vrQoUOqW5sQBdQ0LoeoCxcu6E9/+pPi4+MlXU9vRUcZnDlzRjExMVq1apX++te/qnPnzhUe32w2q7CwsNzjEZxJjdu3b9fq1as1ePBgDR06tMK1AAAAFHEpRF27dk1PPvmkjh49qnr16unee+/VgQMHHLfWvLy81KhRI2VnZ2vcuHGKjY1VUFBQheaoW7eusrOzlZubW2J7Xl6epOsPOS7L+fPn9fLLL6t58+b685//XKEaXOXj41Pqni4AVc+lXJv0eZzjdVhYmBr4WCqxIgDulJiYWGru+CWXQtSaNWt09OhRdenSRUuWLFGTJk00atQoR4hq3769tmzZoqefflr79+/XRx99pBkzZlRoDn9/f2VnZysjI0MtWrS4ob1orqK9UaVZsmSJMjMzFRoaqujo6GJtR44ckSRt3rxZJ0+eVJs2bfTMM89UqM6ymEymW7L3CsCt4WUu/NVrMz/jQDXi7OHgLoWojRs3ysvLS3PnzlWTJk1K7FOvXj3NmzdPgwYNqvAxBNL1b+UlJSUpOTm5xPOckpOTHf3KUrRnKiEhQQkJCSX2SUpKUlJSknr06OHWEAUAAKofl3ZCHjt2TG3atClxheiXgoKC1Lp1a509e7bCc0REREiSNm3adENbVlaW9uzZI4vFol69epU5zptvvqnExMQS/zz++OOSpMmTJysxMVGrVq2qcJ0AAKBmcSlEFRYWlt/p/1gsFkPL3ZGRkQoMDNTOnTu1Zs0ax/W8vDxNnz5dVqtVUVFR8vPzc7TZbDalpKQoJSXF8UgaAAAAd3Lpdl5QUJBOnDihnJwc1atXr9R+WVlZ+vHHH9W6desKz+Ht7a0333xT48ePV3R0tD799FMFBwfr+++/V3p6ujp27Kjnn3++2HvS0tI0ePBgSVJcXNwtO8oAAADUHC6tREVERMhms2nu3Lll9ps1a5YKCgp0//33G5onPDxc69ev16BBg3TmzBlt375d9evX18SJE7Vq1aoyAxwAAMDNYLI7eyxnCS5cuKDBgwfr0qVL6t+/vx566CHFxMQoOTlZX3zxhZKSkrRmzRrt27dPdevW1caNG0s9ebw6SkhIkNVqla+vr0JDQyu7HABucjHXps5/3ux4fWDmQDXkiAOg2nD297dLt/OaNm2qJUuWaOLEidqyZYvi4n4+N2XIkCGSrh+C6evrq/nz59eoAAUAAKo3l59T8Jvf/Eb/+te/9Pjjj6t58+ay2+2OP02bNlVUVJQ+++wz9e7d2x31AgAAeAS3PDuvWbNmeuWVV/TKK6/IarXq8uXL8vX1Vf369d0xPAAAgMdx2wOIi/j6+srX19fdwwIAAHgUHjsOAABgACEKAADAAEIUAACAAYQoAAAAAwhRAAAABhCiAAAADCBEAQAAGECIAgAAMIAQBQAAYAAhCgAAwABCFAAAgAGEKAAAAAMIUQAAAAYQogAAAAwgRAEAABhAiAIAADCAEAUAAGAAIQoAAMAAQhQAAIABhCgAAAADCFEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAAEIUAACAAYQoAAAAAwhRAAAABhCiAAAADCBEAQAAGFCrsgsAgKqmTi0vTenfrthrADUPIQoAKsjbYtbUyPaVXQaASsY/nwAAAAwgRAEAABhAiAIAADCAEAUAAGAAIQoAAMAAQhQAAIABhCgAAAADCFEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAAEIUAACAAbUquwBnHT9+XO+++6727dunCxcuKCAgQA888IAmTJggX1/fCo21fft2rV69WocPH9bly5fVqFEjdevWTU899ZQ6d+58kz4BAACoTqrEStTBgwc1fPhwxcbGys/PT3369JHVatWyZcs0cuRI5eTkOD3W/PnzNWHCBO3cuVNBQUHq06ePGjRooM2bN+t3v/ud/vnPf97ETwIAAKoLj1+Jys/P13PPPSer1arZs2crKipKkpSXl6epU6dq69atmj9/vl577bVyx4qPj9fy5cvl4+Oj5cuXKzw83NG2du1azZw5U6+99prCw8MVGBh40z4TAACo+jx+JWrjxo1KTU1Vz549HQFKkry9vfXGG2/I19dX69at08WLF8sd6x//+Ick6amnnioWoCRp5MiRioiI0LVr17Rp0yb3fggAAFDteHyI2rp1qyQpMjLyhrbGjRsrPDxcNptNO3bsKHcsb29vtW/fXvfcc0+J7XfccYckKS0tzYWKAQBATeDxt/OSkpIkSSEhISW2t23bVtu2bdPRo0c1ZMiQMsf6n//5nzLbDxw4IEkKCAioeKFlsNvtKigocOuYAADg5rDb7U718/gQlZ6eLklq1qxZie3+/v7F+hm1detW/ec//5HFYilx1csVubm52r9/v1vHBAAAlcvjb+dZrVZJ12/FlaToelE/IxITE/Xyyy9Lur5fKigoyPBYAACgZvD4lSiz2azCwkKZTKYy+zm79PZrBw8e1Pjx45Wdna2+ffvq2WefNTROWXx8fEq9HQkAADxLYmKicnNzy+3n8SGqbt26ys7OLvXD5OXlSboeVCrqq6++0ksvvaTc3FwNGDBACxYskJeX+xfnTCaTzGaz28cFAADuV97CTRGPv51XtOcpIyOjxPaivVBF/Zz17rvv6k9/+pNyc3M1atQoLVq0SLVr13atWAAAUGN4fIgqug2WnJxcYnvRdWdvlxUWFuqll17SokWL5OXlpenTp2vmzJmsFAEAgArx+BAVEREhSSUegJmVlaU9e/bIYrGoV69eTo03Y8YM/fOf/5Svr6+WLl2qxx9/3K31AgCAmsHjQ1RkZKQCAwO1c+dOrVmzxnE9Ly9P06dPl9VqVVRUlPz8/BxtNptNKSkpSklJkc1mc1z/7LPP9Omnn6pWrVpasmSJI6ABAABUlMlu9Gttt9CePXs0fvx45eXl6c4771RwcLC+//57paenq2PHjlq1apXq1avn6H/q1Cn1799fkhQXF6fg4GAVFBSof//+Onv2rJo1a6YePXqUOt99992noUOHulx3QkKCrFarfH19FRoa6vJ4AADg5nP297fHfztPksLDw7V+/XrFxMTou+++U3JysoKDgxUVFaUnn3yyWIAqTWJios6ePSvp+mNdYmNjS+3boEEDt4QoAABQfVWJlaiqipUoAACqHmd/f3v8nigAAABPRIgCAAAwgBAFAABgACEKAADAAEIUAACAAYQoAAAAAwhRAAAABhCiAAAADCBEAQAAGECIAgAAMIAQBQAAYAAhCgAAwABCFAAAgAGEKAAAAAMIUQAAAAYQogAAAAwgRAEAABhAiAIAADCAEAUAAGAAIQoAAMAAQhQAAIABhCgAAAADCFEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAAEIUAACAAYQoAAAAAwhRAAAABhCiAAAADCBEAQAAGECIAgAAMIAQBQAAYAAhCgAAwABCFAAAgAGEKAAAAAMIUQAAAAYQogAAAAwgRAEAABhAiAIAADCAEAUAAGAAIQoAAMAAQhQAAIABhCgAAAADCFEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwoFZlF+Cs48eP691339W+fft04cIFBQQE6IEHHtCECRPk6+tbobHS0tK0ZMkS7dq1S+fOnZOfn5/69eunSZMmqUmTJjfpEwAAgOqkSqxEHTx4UMOHD1dsbKz8/PzUp08fWa1WLVu2TCNHjlROTo7TY6WmpurRRx/V2rVr5e3trb59+8psNmv16tUaNmyYzp07dxM/CQAAqC48fiUqPz9fzz33nKxWq2bPnq2oqChJUl5enqZOnaqtW7dq/vz5eu2115wa76WXXlJGRoYmTZqkZ599VpJUUFCg6OhorV27VjNnztTy5cvd+hkKCu26mGtz65gAAODmKCi0O9XPZLfbnetZST7//HO9+OKL6tmzp1asWFGsLSsrS/369ZPNZtO3336rhg0bljlWfHy8Ro8erdatW+vLL7+Ul9fPC3E2m00DBw7UmTNntHHjRrVt29bl2hMSEmS1WpWSZdOLWy64PB4AALj53h7QVG0aW+Tr66vQ0NBS+3n87bytW7dKkiIjI29oa9y4scLDw2Wz2bRjxw6nx+rfv3+xACVJFotF/fr1kyTFxcW5WjYAAKjmPD5EJSUlSZJCQkJKbC9aMTp69KjTY7Vv397lsQAAQM3m8Xui0tPTJUnNmjUrsd3f379Yv1s1VkWEBtTX9zO6uHVMAABwc5w6kaJrV/PK7efxIcpqtUqSvL29S2wvul7Uz5mxfHx8XB6rIq5dzdOxxCNuHRMAAFQuj7+dZzabJUkmk6nMfs7sj3fnWAAAoGbz+JWounXrKjs7W7m5uSW25+VdX24rbXXp12NJcstYFeHj41Pqni4AAOBZEhMTS80Kv+TxIcrf31/Z2dnKyMhQixYtbmgv2r9UtJ+pvLGOHDmijIyMEtsrMlZFmEwmxyoYAADwbOXdsSri8bfzilZwkpOTS2wvuu7MSo87xwIAADWbx4eoiIgISdKmTZtuaMvKytKePXtksVjUq1cvp8f6+uuvVVhYWKzNZrM5zofq27evq2UDAIBqzuNDVGRkpAIDA7Vz506tWbPGcT0vL0/Tp0+X1WpVVFSU/Pz8HG02m00pKSlKSUmRzfbz41a6deumsLAwpaSkaP78+Y4N5AUFBZo9e7bOnj2r3r17l3k6KQAAgFQFHvsiSXv27NH48eOVl5enO++8U8HBwfr++++Vnp6ujh07atWqVapXr56j/6lTp9S/f39J108fDw4OdrT9+OOPGjNmjLKzs3XHHXeoXbt2SkhI0E8//aSgoCB98sknpZ4jVVFFj30p79h4AADgOZz9/e3xK1GSFB4ervXr12vQoEE6c+aMtm/frvr162vixIk3BKjytGvXThs2bNDw4cN1+fJlbdu2TZI0duxYrVu3zm0BCgAAVG9VYiWqqmIlCgCAqqdarUQBAAB4GkIUAACAAYQoAAAAAzz+xPKq7OrVq5KuP2YmISGhkqsBAADOKHrkS9Hv8dIQom6iogM97Xa7rFZrJVcDAAAq4tcHc/8aIeomslgsstls8vLyUp06dSq7HAAA4ISrV6+qsLBQFoulzH4ccQAAAGAAG8sBAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAAEIUAACAAYQoAAAAAwhRAAAABhCiAAAADCBEAQAAGECIAgAAMIAQBQAAYAAhCgAAwABCFAAAgAGEKAAAAAMIUQA80uLFixUSEqLo6Gin35Obm6slS5bokUceUZcuXXTXXXfpgQce0Lx585SdnV2h+ffs2aOQkBANGTKkgpUXd+LECXXp0qVCn8NZaWlpmjlzpiIjIxUWFqa+ffvq9ddfV2ZmptvnAnAjQhSAaiE7O1sjRozQO++8o9OnT6tr167q0aOHMjMz9d5772n48OE6d+7cLa3p/PnzmjhxonJzc90+dmpqqh599FGtXbtW3t7e6tu3r8xms1avXq1hw4bd8s8K1ESEKADVwty5c5WUlKQePXpo8+bN+uijj/T+++/r66+/1v3336/Tp0/r1VdfvWX1JCQkaNSoUUpJSbkp47/00kvKyMjQpEmTFBsbq0WLFmnTpk0aOXKkzp07p5kzZ96UeQH8jBAFoMrLy8vTF198IUl688031aRJE0dbgwYN9NZbb8lkMmnHjh3Kysq6qbVcvHhRc+fO1YgRI3Ty5EkFBwe7fY74+HjFx8erdevWmjx5suO62WzWjBkzFBgYqO3btys5OdntcwP4Wa3KLgAAyrNr1y4tXrxYP/zwg7y9vXX33XdrwoQJCgsLkyRduHBBd955p+x2u4KCgm54f9OmTdWwYUNlZ2crPT1djRs3rtD8P/30k+bNm6fdu3crPz9foaGhGjNmjAYPHnxD35UrV+r9999XQECAZs6cqSNHjigmJqbUsUNCQpyqISgoSFu3bpUkx//2799fXl7F/y1ssVjUr18/rV69WnFxcWrbtq2zHxNABRGiAHi0Xbt26ZNPPlFAQIAiIiKUmpqqr7/+Wtu2bdPChQsVGRmpoKAg/e1vfyt1jJMnTyo7O1teXl5q1qxZhea/cOGCRowYIZvNpvDwcF25ckV79+7Vvn37dOjQIU2bNq1Y/4CAAE2bNk2jRo2St7e3jhw5Uub4Dz30kFN1/HJ1LSkpSZLUvn37EvsWBaejR486NTYAYwhRADza8ePHNWzYMEVHR6t27dqSpDVr1ig6OlqvvPKKunfvXu7K0rx58yRJ9957rxo1alSh+TMzMxUaGqoPPvhATZs2lSTt379fTz75pD788EP17t1bPXv2dPR/7LHHKjR+UW0VkZ6eLkmlBkJ/f/9i/QDcHOyJAuDRmjZtqldffdURoCRp9OjRioiI0KVLl/T555+X+f5ly5Zp8+bN8vb21osvvmiohujoaEeAkqQuXbromWeekXT99t2tZrVaJUk+Pj4ltnt7exfrB+DmIEQB8GiRkZGqW7fuDdcHDBggSfr3v/9d6nsXLVqkBQsWyMvLS2+88YbT+49+6Y477tBdd91V6vzfffddhcd0ldlsliSZTKYy+9nt9ltRDlBjcTsPgEdr0aJFidebN28uqeRbVteuXdOMGTP0+eefq1atWpozZ44efPDBYn2WLl1a4vEDAwcO1MCBA8udPzAwUJKUk5OjnJwc1atXz7kP9CtGNpYXhcrSzp/Ky8uTVPpKFQD3IEQB8Gh16tQp8XrRKovFYil2/cKFC5o0aZK+//571atXT++8847uu+++G96/a9euEleRWrVqVSxElTb/L/3yVmNFGdlY7u/vryNHjigjI6PEvkXBsmhvFICbgxAFwKOlpaWVeP3UqVOSfl6Rkq4fRfDEE0/o9OnTCg4O1rJly9SuXbsS379q1SqX5k9NTZV0fc+WKyHKyMbykJAQbdu2rdRzoIquG7l9CcB57IkC4NF27NhR4t6eL7/8UpJ0zz33SLoedh5//HGdPn1ad911l9atW1dqgKqIH374QefPny93/lspIiJCkvT111+rsLCwWJvNZlNcXJwkqW/fvre8NqAmIUQB8GhHjx7VwoULiwWp5cuX67vvvlOzZs308MMPS5JeeOEFnT17ViEhIVqxYkWxb9O5wmaz6cUXX9SVK1cc13bv3q333ntPZrNZf/jDH9wyT0V069ZNYWFhSklJ0fz58x3/bQoKCjR79mydPXtWvXv3Vmho6C2vDahJuJ0HwKN17dpVy5Yt06ZNmxQSEqLk5GQlJyerfv36WrRokXx9ffXtt99qz549kqT69euX+dy4KVOmlLpZvCTt27fX/v37NWDAAHXv3l1ZWVmKj4+XJL366qvq1KmTax/QoDlz5mjMmDF67733FBcXp3bt2ikhIUE//fSTgoKCNGvWrEqpC6hJCFEAPNqQIUM0btw4LV++XFu3blW9evX0yCOP6I9//KMjDG3bts3RvyjglOb3v/99hULU7bffrrlz52ru3LnauXOnTCaTevbsqQkTJlTKrbwi7dq104YNGxQTE6MdO3Zo27ZtCggI0NixY/Vf//Vf8vPzq7TagJrCZOcgEQAAgApjTxQAAIABhCgAAAADCFEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAAEIUAACAAYQoANVebm6uTp06VdllAKhmCFEAqrXY2FgNGjRIu3fvruxSAFQzhCgA1dqCBQuUlpZW2WUAqIYIUQAAAAYQogAAAAwgRAGolhYvXqyQkBCdPn1akjRjxgyFhIRo8eLFjj7nz5/X22+/rcGDB6tz587q2rWrHn30UX344Ye6evVqqWPOmzdPW7Zs0aBBg9SpUyf169dPGzdu1KlTpxQSEqLevXursLBQa9as0dChQ9W5c2fdc889mjRpklJSUiRJmZmZev3119WnTx916tRJERERio6O1uXLl2/NfyAALqtV2QUAwM3QvHlzdevWTYcPH9a1a9fUqlUrNW3aVM2bN5ck7du3TxMnTlR2drYsFotat24tu92uI0eO6PDhw/r888/1/vvv67bbbrth7L179+rDDz9Uw4YN1aZNG6WkpCg0NNTRXlhYqClTpmjz5s1q1qyZWrVqpWPHjmnLli3au3evli9frilTpigjI0OtWrVSYGCgTp48qTVr1uiHH37QJ598IpPJdMv+WwEwxmS32+2VXQQA3Cz9+vXT6dOnNWvWLD322GOSpLS0ND388MPKzs7WiBEj9MILL6hBgwaSpJ9++knPP/+8Dhw4oO7du2vNmjWOsRYvXqyYmBhJUmRkpObPn6/atWsrMzNTTZo00alTp9S/f39JUq1atTRr1iwNHTpUJpNJSUlJGjFihHJzc+Xl5aUOHTpowYIFat26tSTp008/1SuvvCJJWrVqlXr06HGr/hMBMIjbeQBqnA8++EDZ2dnq16+fXn/9dUeAkqSWLVtqyZIlqlevnuLj4/W///u/JY4xbdo01a5dW5LUpEmTG9qjoqI0bNgwx4pS+/btHQHLbrfrnXfecQQoSXr00UcVFBQkSfrhhx/c8jkB3FyEKAA1zpYtWyRJDz/8cIntfn5+6tWrlyRp27ZtN7TfdtttatGiRZlz9OnT54ZrRSHp9ttvV8uWLW9o9/f3lyTl5OSUOTYAz8CeKAA1ypUrVxybzZcsWaKVK1eW2K+oz7Fjx25oKwo7ZSnae/VLFotFUskrV79sZ5cFUDUQogDUKL9c5UlKSiq3f0nflqtTp0657/Px8Sm1zcuLmwBAdUCIAlCj/DLcxMbGqn379pVYDYCqjH8OAahRGjRoID8/P0lScnJyqf0SExOVkJCgixcv3qrSAFQxhCgA1VrRt+N+uc+oaNP36tWrVVhYeMN7Ll++rN///vcaOnSoPv7441tSJ4CqhxAFoFrz9fWV9PNGcUkaP368fH19tW/fPr3wwgvKzMx0tJ0+fVrjx49XVlaW6tevr9GjR9/ymgFUDeyJAlCtdezYUUlJSXr//ff1zTffKDIyUhMnTtTChQs1depUffHFF9q0aZPatm0rm82mEydOKD8/X76+vvrrX/+qpk2bVvZHAOChCFEAqrVp06YpNzdXu3bt0rFjxxzProuIiNDGjRu1YsUK7dixQ8ePH1dBQYGCgoLUq1cv/eEPfyj3LCgANRuPfQEAADCAPVEAAAAGEKIAAAAMIEQBAAAYQIgCAAAwgBAFAABgACEKAADAAEIUAACAAYQoAAAAAwhRAAAABhCiAAAADCBEAQAAGECIAgAAMIAQBQAAYMD/B3hKnyFjtN3rAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Plot\n",
"ax = sns.pointplot(data=eq, x='term', y='estimate')\n",
"ax.vlines(eq['term'], eq['conf_low'], eq['conf_high'])\n",
"ax.axhline(0.01)\n",
"ax.axhline(1)"
]
},
{
"cell_type": "markdown",
"id": "c6643063-27e2-40d3-9633-9189b58e528e",
"metadata": {},
"source": [
"To reiterate, this test is significant, which means you can say the effect is *unintersting, null, or too small to be of interest*. If the test was *not* significant, you would conclude that the intervention on your looks could possibly yield effects outside of this interval and thus *could be of interest*."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
}
},
"nbformat": 4,
"nbformat_minor": 5
}